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ABSTRACT 

The current philosophical position of the project is 
reviewed and a variety of issues relevant to educational evaluation 
and achievement monitoring are enumerated. Various past attempts to 
model the variables of school learning are reviewed and those adopted 
for the Comprehensive Achievement Monitoring model are discussed. The 
student variables include verbal, reasoning, spatial, perceptual, and 
memory aptitudes in the genera) domain; and prior achievement., 
motivation, anxiety, sociological characteristics, and rate of 
learning in the task specific domain. Instructional variables are 
teacher, treatment characteristics, and time. Two important features 
of the program are (1) the emphasis placed on defining course 
objectives in a detailed manner, and (2) monitoring student 
achievement on these objectives longitudinally. The report also 
discusses the focus, resources, collection, organization, and 
analysis of data that will be used to evaluate the project. Finally, 
the favorable teacher and student reactions to the monitoring are 
discussed. (AE) 
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Abstract 

The Second Annual Report to the Charles F. Kettering Founda- 
tion on the progress to develop a comprehensive randomized monitor- 
ing is designed to outline the experience and flexibility that the 
project has gained from its work. The report includes an overview 
of the current philosophical position of the project. A variety 
of issues relevant to educational evaluation and achievement monitor- 
ing are enumerated. Tho report closes with a discussion of the 
teacher and student reactions to the monitoring and the priorities 
for the future work of the staff. 
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Overview 

Need for Achievement Monitoring 

There has rarely been a time in the history of American 
education when there has been such an intense concern for educa- 
tional evaluation. Many books discuss evaluation in detail, 
including a revision of Educational Measurement and the next year- 
book by the NSSE. The pages of magazines such as Educationa l 
Technology , Educational and Psychological Measurement , and Journal 
of Educational Measurement , are bulging with articles delimiting 
the need for evaluation in school courses. The association for 
Supervision and Curriculum Development has devoted an entire volume 
of its recent yearly publication to evaluation as feedback and 
guide (195?)* Further, a very important article by Cronbach (1963) 
entitled "Evaluation for Course Improvement' 1 is becoming more widely 
read and quoted. Much of the recent American Educational Research 
Association monograph, editing by Stake (196?) entitled "Curriculum 
Evaluation" is concerned with specifying in more detail issues 
raised by Cronbach. 

There seems to be no subject or grade levwl in public 
schools which can reasonably ignore evaluation. It is important 
for every school course to consider more seriously its objectives 
and how well these objectives are being reached. Our project 
focuses on an important aspect of evaluation, achievement monitor- 
ing. It has considered a variety of subjects and grade levels as 
shown in Table AR-2.1. 

Comprehensive Achievement monitori ng 

One of the primary objections the current authors raise to 
the earlier methodology of evaluation is its narrowness. Many of 
the early evaluations or those of Title I and Title 111 projects 
consists of unsystematic interviews with teachers and students and 
a posttest of student achievement using standardized batteries of 
tests. These techniques make it difficult to pinpoint the 
strengths and weaknesses of a course if only one or two measures 




Table AR-2.1 Comprehensive Achievement Monitoring: 
Subjeot and Grade Level 



Subjeot 


a 


r a 


d o 


1 e 


v e 1 


Pre-high 

sohool 


9 


10 


11 


Post-high 
xc sohool 


Soienoe 


P 


C 


c 


P 




Mathematics 


P 


c 


c 


c 


C C 


English 






0 






History 


P 




c 


c 


C 


Vocational 










C 



Note.--C & operational and P » planned soon for 
Comprehensive Achievement Monitoring. 
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are taken or if the measures focus on only some general and nebulous 
quantity called achievement. 

Our project has sought to develop a comprehensive design 
for the monitoring of student achievement in high school courses. 

The achievement monitoring design developed by the project is com- 
prehensive in two primary dimensions. It is comprehensive in 
terms of a multivariable model of school achievement and the longi- 
tudinal measurement of the objectives of a course. 

A Model of Sohool Achievement 

Several recent efforts have attempted to model school 
learning. Carroll (1967) has described a semi-quantitative model 
which was used by Bloom (in press) to describe rates of student 
learning. Carroll provides a basis to decide which variables to 
measure in an attempt to adequately describe sohool learning. 

The project accepts the very important notion that a model 
of sohool learning or more specifically a model of sohool achieve- 
ment is necessary to focus an evaluation design on the relevant 
variables. However t the design suggested by Carroll seems to 
ignore several variables or interactions of variables which seem 
to be particularly relevant for school achievement. 

First) Carroll indicates that a measure of perseverance 
is an important contributor to the model of school achievement. 

The notion of perseverance seems to lock adequate ties with 
earlier and better founded notions of motivation which can them- 
selves be used as variables in a model of school achievement. 

Our project prefers to use the better documented notions of 
achievement motive (Alschuler). 

Second) Alschuler explains that although a student may be 
highly motivated to perform a particular task) his anxiety at 
entering task situation may prevent or interfere with several of 
the skills whioh are necessary in performing the task. The effeot 
of anxiety on complex learning has been investigated by Spielberger 
(1966) and it eee&s relevant to the model of school achievement. 

This is particularly evident in schools where students are under 
very high parental or peer pressure to succeed academically . 
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Third, sociological variables appear important. Coleman 
(1966) has demonstrated a correlation of certain social and family 
variables with school achievement. They seem important particularly 
in public schools which enroll students from a broad range of 
family backgrounds to include a variety of measures of student 
family community and peer influences on the student. 

Fourth are the variables of rate of learning and initial 
aohievement (Yeager & Lindvall, 19o7) . In the school environment 
the rate of learning and initial achievement seem to correlato very 
highly with achievement output. It may also be relevant to follow 
the variation in rate with the other student variables already 
mentioned (Tucker, I960) . 

Although Carroll mentions both instructional and student 
variables, he does not explore the implications of their inter- 
actions as clearly as Cronbach (196?) has. It is very important 
to understand that the interactions of instructional treatment 
and student variables may drastically change the interpretation 
of results for student achievement and must be considered explicitly 
in a model of school achievement. 

The constituents of a model of student achievement whioh 
the project Intends to utilise are outlined in school achievement 
in Table AR-2.2. 

A further consideration in evaluation of school achievement 
is suggested by itothkopf ( 1965 ). He describes how the evaluation 
may interact with student achievement. Some forms of evaluation 
may increase students' learning. Rothkopf has shown that certain 
kinds of tests administered during learning, from written mate- 
rials, increases achievement. The project, in an experiment with 
videotaped lectures (Tii-4),' has shown a similar effect on stu- 
dents' achievement with different patterns of comprehensive 
pretesting. 

Thus, some experimental evidence exists that oven the form 
of the evaluation of achievement should be entered into the oodel 
of sohool achievement. Experimentation in high school courses by 




*TM indicates a project Technical Memorandum. 



Table AR-2.2 Comprehensive Achievement Monitoring: 





The Model 


of School 


Achievement 








Instructional variables 






Student 

variable 








Domain 


Teacher 


Treatment 

characteristics 


Time 




Aptitude : 










Verbal 


— 


— 


-- 


General 


Reasoning 

Spatial 


— 


-- 


— 


- “ 




— — 




Perceptual 


— 


— 


-- 




Memory 


— 


-- 


-- 




Prior achievement 


-- 


«... 






Motivation 




- - 


- - 


Task 

specific 


Anxiety 




— 


— 




Sociological 










characteristics 










Rate of learning 
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t.ie project (TM-16) suggests that these effects ere very small 
and therefore are not included in the model. 

Objectives and Time: Important 

Treatment Characteristics 

It is important to use a model of school achievement to 
suggest variables which are relevant for an achievement monitoring 
program. Two important features of the comprehensive achievement 
monitoring are (a) the emphasis on defining the objectives of the 
course in a detailed fashion, and (b) monitoring the students' 
achievement on these objectives longitudinally. 

It has only been in reoent years that teachers have begun 
to define their objectives in behavioral terms. Authors like 
Mager (1962) have written persuasively in favor of specifying 
instructional objectives in terms of observable student behavior. 
In the case of comprehensive achievement monitoring, the behavior 
desired is performance on achievement questions. The teacher is 
expected to define the objectives of his course in terms of 
observable student behavior by composing questions which measure 
acceptable levels of student performance for the objectives he 
set for his course. 

A second unique aspect of the comprehensive achievement 
monitoring is the longitudinal consideration of students achieve- 
ment on each of the objectives of a course. A comparison of the 
estimates of achievement available by che usual classroom testing 
(Table AR-2. 3) and comprehensive achievement monitoring (Table AR- 
2.4) highlights the characteristics of longitudinal monitoring. 

In usual classroom testing a test of achievement is given imme- 
diately after an objective or set of objectives has been presented 
by the teacher. The teBt usually includes items which measure 
only the objectives taught since the last testing. Therefore, as 
Table AR-2. 3 shows, the acher has available only estimates of 
student achievement on the objectives he has just completed 
teaching. 




Table AR-2,3 Usual Classroom Testing: Achievement 

for a Specified Group of Students for 
Objectives by Time 



Objective 






Time 




f 


1 


2 


3 4 


• f » 


T 


1 


u 










2 




U 








3 






U 






4 






U 






t • • 








U 




0 










U 



Note.--U = estimate of achievement available by 
usual classroom testing. 



Table AR-2.4 Comprehensive Achievement Monitoring: 
Achievement for a Speoified Group of 
Students for Objectives by Time 



Objective 






T 


i ffl e 






1 


2 


3 


4 


• » • 


T 


1 


c 


c 


c 


C 


c 


c 


2 


c 


c 


c 


C 


c 


c 


3 


c 


c 


c 


c 


c 


c 


4 


c 


c 


c 


c 


c 


c 


• • • 


c 


c 


c 


c- 


c 


c 


0 


c 


c 


c 


c 


c 


c 



Note. — C = estimate of achievement available by 
Comprehensive Achievement Testing. 
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Contrasted with the usual testing is comprehensive achievement 
monitoring where estimates of student achievement on each of the 
objectives &oe available throughout the time of monitoring. This 
type of testing allows teachers to make statements about a stu- 
dent’s rate of learning, achievement immediately after instruction, 
and rate of forgetting. For example, consider achievement moni- 
tored at time 4. If time 4 is immediately after objective 4 has 
been taught, the estimate of student achievement for objective 4 
is an immediate posttreatment achievement. At time 4 the esti- 
mates of achievement on objectives 1, 2, and 3 represent achieve- 
ment a given time after the teaching of these objectives and 
measure forgetting. Further, the achievement on objectives to be 
taught next in the course are also measured by the comprehensive 
achievement monitoring. Therefore, preins true ttonal treatment 
estimates of achievement are available to the teacher. The impor- 
tance of the availability of these estimates on all of the 
objectives for a course at each time will be discussed below. 
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A Design for Comprehensive 
Achievement Monitoring 

Although the model of student achievement suggests the 
variables which are important for the consideration of achievement, 
a design must be developed which operationalizes the measurement 
of these variables in the school. 

The Matrix of a Design 

A variety of authors have suggested designs for educational 
evaluation. Most of these designs are much more concerned with the 
operational needs of an evaluation system rather than the model 
from which the evaluation is developed. We will add the notion of 
a model of school achievement to the operational considerations 
discussed by several designs for educational evaluation (Stuffle- 
beam, 1968). 

The matrix of a design for comprehensive achievement moni- 
toring is displayed in Table AR-2. 5 • The domains of the me litoring 
may be considered to include the context, input, process and 
product of learning* We will consider context of the monitoring 
to include measures of the students who enter a program of study 
concerned with their general level and abilities. The input will 
be considered the achievement and knowledge which they bring to 
the learning situation. The process will be the mental processes 
by which they learn the materials of the course and the product 
will be the final behavior in the course. A variety of issues in 
the design of the monitoring program are outlined in the left-hand 
column of the table. Each of the issues in the design will be 
considered below. 



Model of the Parameters 

Student Variables 

During the first year of the project student variables 
were not measured for all tho variables included in the current 
model of student achievement. During the first year a pretest 



Table AR-2.5 The Matrix of a Design for Comprehensive 
Achievement Monitoring 



Issues in design 


Domains of the monitoring 




Context 


Input 


Process Prod’ ot 


Model of the parameters 


C 


C 


P 


C 


Focus of the evaluation 


C 


C 


P 


C 


Resources for the evaluation 


C 


C 


P 


C 


Collection of the data 


C 


C 


F 


c 


Organization of the data 


C 


C 


P 


c 


Analysis of the data 


C 


C 


P 


c 


Report of the analysis 


C 


C 


P 


c 



Note.--C = operational and P = planned soon for Comprehensive 
Achievement Monitoring. 
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or input level of achievement for the course monitored by the 
project was the major pretroatraent measure. This year, following 
the model, measures of student aptitudes, achievement motivation, 
anxiety, demographic information, ani attitude toward mathematics 
were measured by a student pretest and questionnaire (TM-l8). 

Instructional Variables 

The instructional variables recorded include the complete 
list of the objectives for each course specified in behavioral 
terms. Several characteristics of the teachers collected by a 
questionnaire (TM-l8) and measures of the time taken by the 
students to complete certain objectives of the course. 

Focus of the Evaluation 

The leve 1 of decision making . The monitoring of achievement 
can be of interest to the student enrolled in the course, to the 
teacher teaching the course, to the department in which the course 
is taught, to the school, to the school district, to the state and 
even to the national level. Each of these levels would be 
extremely interesting to consider by the evaluation design dis- 
cussed here, but the primary interest of the project has been at 
the student, teacher and departmental levels. The project may 
become involved with school and school district concerns with 
achievement monitoring but these levels will be pursued to better 
provide information for students, teachers and department. At 
each level of evaluation a variety of decision-situations occur. 

The decision-situation . Each decision-situation includes 
a question of the setting (the course in which decisions are to be 
made), the alternatives (choices available), the variables (parame- 
ters of the choice), and the criteria (rules to make the choice 
based upon information collected). A variety of decision-situations 
have been identified by the project. The major decision-situations 
are those of assigning students to different instructional treat- 
ments (TM-2; Silberman, 1968), of encouraging students in their 
studies, and providing information to teachers and departments 
for improving their courses, (see Table AR-2. 6). 






Table AR-2.6 Focus of the Evaluation: Level and 

Decision Situation 



Level 




Decision 


situation 




Setting 


Alternatives 


Variables 


Criteria 


Student 


C 


C 


C 


C 


Teacher 


C 


c 


C 


C 


Department 


C 


c 


C 


C 


School 


P 


p 


P 


P 


Sohool district 

State 

National 


? 


p 


P 


P 



Note. — C “ operational and F = planned soon for Comprehensive 
Achievement Monitoring. 
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Resources for the Evaluation 

Staff . TIk project has considered a variety of patterns 
of staffing for the monitoring of achievement. The job of speci- 
fying a curriculum in behavioral tevms and writing questions to 
measure these objectives is substantial. The best strategy seems 
to be for a team of teachers to work together or to develop an 
objectives item bank. The project has utilized a single teacher 
for a course at a school, a team of teachers and an entire depart- 
ment of a school involved in the monitoring program. 

Facilities . A large range exists in the types of facili- 
ties available to teachers working with comprehensive achievement 
monitoring. Most schools have teaching aid available to assist 
teachers in clerical work and administering monitors. One school 
has a testing room, where students could be scheduled for a 
monitor. Ths teachers working closely with the project have had 
the answer sheets from their monitoring scored by optical scanners 
at Stanford University. Several school districts in the project 
now have optical scanners of their own and are considering 
utilizing a school base facility. During the first year of opera- 
tion all schools sent their monitoring data to Stanford University 
where they were processed and analyzed on the Stanford computer. 
Several schools are using a local data processing system for their 
projects. ( see Table AR-2.7). 

Collection of the Data 

Source of data . The major source of data remains the 
student responses to achievement test questionsl The model of 
school achievement has suggested that other measures of each 
student be taken. Also, information about the teachers and the 
schools are being collected this year. All types of Information, 
in addition to the student responses to achievement items, are 
new sources of data this year. 

Objectives and time . At the heart of the project are the 
variables of objectives and time. Course objectives have been 
all specified in behavioral terms. The specificity of the 



0 



Table AE-2.7 Resources for the Svaluation: 
Staff and Facilities 



Staff 




F a c i 1 


i t 1 e e 




Teaching 

aids 


Testing 

room 


Optical 

scanner 


Computer 


One teacher 


C 




C 


C 


Team of teachers 


C 




C 


C 


Department in school 


C 


C 


c 


c 


Faculty of school 




P 


p 


p 


School district 






p 


p 


Private agency 










State officials 










National officials 











Note. — C = operational and P = planned soon for Comprehensive 
Achievement Monitoring. 
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objectives does vary. In some cases the objectives are written 
for individual tasks. In other cases a lesson which might take 
one d' to present is the unit of specificity. Other levels 
include a package or unit which includes several objectives which 
are interrelated and are presented to students within a period of 
about a week. Some objectives for a course or a school program 
have been discussed. The specificity of objectives varies from 
one school to another. The project is considering the most appro- 
priate level of specificity for different types of achievement 
monitoring. 

A taxonomy of educational objectives has been suggested by 
Bloom (1956). This taxonomy has been considered as a way of cate- 
gorizing the objectives which have been written for the project. 

One school has been instructed to classify its mathematics objec- 
tives both in terms of a level of cognitive difficulty and in 
terms of mathematical content (TM-2). A variety of other schemes 
may be tried. The important consideration will be to provide an 
operational technique for categorizing objectives into meaningful 
headings in order to devise better ways of analyzing results. A 
booklet of the objectives written for the project will be developed 
so that teachers preparing courses in subjects already monitored 
will be able to compare their suggested objectives with an already 
written set of objectives. This may reduce one of the bottlenecks 
for instituting comprehensive achievement monitoring. 

The other major dimension, time, is specified by the inter- 
val between monitoring of student achievement. The project has 
tried a variety of time intervals. Table AR-2. 8 displays the time 
intervals used as well as the specificity of performance objectives 
The time interval is very closely related to the logistics monitor- 
ing and the need for current up-to-date information. The balance 
between these two factors and that of time has been explored and 
several alternatives may be suggested. 

Sampling of students and items . Important dimensions of 
the project are objectives and time, but the technique used to 
obtain estimates of achievement on each objective at each time is 



O 
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Table AR-2,8 Collection of Data: Performance Objectives 

and Time — the Major Dimensions 



Specificity of 




Time 


interval 


between 


monitorings 




performance 

objectives 


Day 


Week 


Two 

weeks 


Month 


Semester 


Year 


Individual task 


P 


C 


C 








Lesson 


P 


C 


C 








Package 
or units 


C 


C 


C 


C 






Course 




C 


c 


P 


P 


P 


School program 








P 


P 


P 



tfote.--C e operational and P » planned soon for Comprehensive 
Achievement Monitoring* 
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the technique of sampling. The students and items form two pools from 
which samples may be drawn Id obtain estimates of achievement. The 
technique of sampling items and students has been suggested by Cronbach 
1963) sod the theory of the sampling has been detailed by Lord and 
Hovick (3968). A large variety of different techniques have been uti- 
lized by the project in its past and current monitoring programs. 

Table AR-2. 9 displays the type samples possible from the pool of items 
and students. A variety of prescriptions have been developed to aid both 
research questions, course evaluation and student feedback to make them 
as effective as possible. 

tfith the development of such a large number of items the project 
has considered developing an item bank. An item bank would provide a 
ready access of items to measure specific objectives for teachers 
developing comprehensive achievement monitoring programs. The items 
could eventually be standardized in terms of the students who take the 
item monitors based upon their performance as well as other variables 
in the model. Thus an item may have a norm related to students with 
certain cognitive abilities from certain types of socioeconomic back- 
grounds who have taken the item at particular times during the school 
year or before or after the item has been taught. 

Instruments . The collection of data on certaiu variables for 
the model of school achievement has been accomplished at the beginning 
of the year with an extensive set of pretests (TM-18). During the 
school year the achievement monitors administered by the program have 
been ohosen to be parallel in content. The additional important 
restriction has been placed on the selection of items for monitors 
during the second year of the project t i.e., that the monitors be some- 
what equivalent in difficulty. The technique of constructing indi- 
vidual teats, which are equal in difficulty, allows comparisons of 
total test scores from one time period to another for individual 
students (TM-15) and is a unique feature of the comprehensive monitoring 
program. 

Schedules . The logistics of achievement monitoring can become 
quite complicated. The basic design of the comprehensive monitoring 
program in general necessitates the administration of all of the items 




Table AR-2.9 Collection of Data: Sampling of Students and Itemsi 

for Each Performance Objective at Each Time 



Sample 

number 



Item sample (n.) 

of pool (N) 1 
for eaoh objeotive 



Student sample of pool 
1 2 3 4 5 6 



1 One sample 

<n ± < N) 

2 Equal samples, 

nonexhaustive 
(hj. * n.| £n i < N) 

3 Unequal samples, 

nonexhaustive 
(n t 4 n^j £ n i < H) 

4 Equal samples, 

exhaustive 

(n^ b f Dj a N) 

5 Unequal samples, 

exhaustive 

(n A / n^ j £ ■ N) 



c c c c c c 



c c 



c c 



c c 



c c 



6 Total pool (if) 



C C C 



Hote .» -Criteria for sampling items or students may be a funotlon 
of time during the year. C » operational and i * planned 
soon for Comprehensive Achievement Monitoring. 
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selected for the Initial pool to at least some of the students every 
monitoring period . Therefore, all of the parallel monitor forms con- 
structed at the beginning of the year are administered each monitoring 
period* A variety of techniques has been investigated for simplifying 
the presentation of monitoring schedules and to facilitate teacher 
administration of the monitors. A viable set of suggestions has been 
developed for both teacher and educational research monitoring. 

Organization of the Data 

Monitor generation . Because of the large number of test ques- 
tions used in the achievement monitoring program, various techniques 
were developed to generate the monitor forms needed. These techniques 
are outlined in the left-hand column of Table AR-2. 10. The projoct has 
utilized the first three techniques (TM-14), Suggestions for tests 
generated by algorithms, by computer, or presented by computer, 
linear or branched, are possibilities for future applications in com- 
prehensive achievement monitoring (Harmand, Helm, & Loye, 1968). 
Although these would be interesting and possible profitable directions 
for the monitoring procedure, they are much more expensive and require 
more computer hardware and software than is available right now in 
many schools. The direction of the project for the rest of this year 
and the coming year will probably be in terms of teacher generated 
monitors or inexpensively computer generated monitors. 

Response coding . Another major consideration in the organiza- 
tion of data is the technique used for coding the responses of students 
in a form that can be analyzed. A large variety of techniques have 
been tried for this purpose and they are displayed in Table AR-2. 10. 
Optical scanner or test scores are becoming increasingly available in 
schools. Their usefulness in reducing the burden of coding responses 
cannot be underestimated. The usefulness of types of machines for 
response coding by the project will be explored and written up. 

Data storage and retrieval . Because the project is working 
with about 1500 school students in nine different courses across the 
country (TM-17) an efficient and versatile computer program was 
developed to edit, collate, and Btore all of the data collected about 




Table *£-2.10 Organization of the Data: Monitor 

Generation and Response Coding 
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students and their responses to test questions. This program is 
described in detail in TM-7. 

Analysis of the Data 

Frequency of analysis . The frequency with which the analyses 
are performed is dependent very much on the types of decisions to be 
made. Decisions about student performance or instructional treatment 
must be made frequentlyi at least every monitoring period. Decisions 
about course improvement or experimental considerations can be made 
once or twice during the course. The frequencies of analysis are 
outlined in Table AR-2. 11. 

Type of analysis . A large range of analyses has been utilized 
to better understand school achievement of students. The model of 
school achievement suggests certain kinds of analyses. Others are 
important to teachers to analyze courses or to the achievement monitor 
ing program in general. The analyses include information about indi- 
vidual student’s performance, overtime , the performance of the class 
in general, performance of the student on specific objectives, the 
quality of individual items, the achievement of different objectives 
during the year, the reliability of questions and test scores, and the 
effects of certain attitudes towards the subject or the course in 
student's achievement The model itself suggests certain questions 
of the relevancy of different variables to school achievement for 
different subjects for instructional tasks and some of each kind have 
been utilized fcr one or more components of the comprehensive achieve- 
ment monitoring, as detailed in TM-6, TH-10, TM-15, and ltt-16. 

Means of analysis . The teachers have been able to perform a 
variety of analyses of students, classes, objectives, and achievements 
by hand. For more complex analyses, the Stanford Computation Center 
has served to run programs written specifically for the project to 
analyze longitudinal comprehensive achievement monitoring data (TM-ll, 
1M-12, TM-14), and the library computer programs have been used for 
certain kinds of analyses of learning models and experiments (TM-8, 
TM-16). 
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Table AR-2.11 Analysis of the Data: Frequency, 

Type, and Means of Analysis 









Means of analysis 


Frequency 
of analysis 


Type of 
analysis 


Teacher 

or 


Project 

computer 


Library 

computer 






staff 


programs 


programs 




Student 


C 


C 




Monitoring 

period 


Class 


C 


C 






Objective 


C 


C 






Item 




C 






Achieveaent 


C 


C 




Course 


Reliability 




C 






Attitude 


C 


C 






ParaneterB 




C 


C 


Experiment 


Models 




C 


C 




Covariance 






C 



Note.-~C * operational for Comprehensive Achieveaent Monitoring 
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Report of the Analysis 

Audience . The reports of the analyses are currently being 
distributed to the students and teachers of the course and being con- 
sidered by the research staff of the project. 

Means of reporting . Efficient! rapid, and comprehensive report- 
ing is needed to complete the cycle of the evaluation. The finest data 
will have no impact on the educational program of a school if they are 
not readily available. 

The report of the analysis is pri ted by the computer in a form 
which can be easily read by teachers, students, and the project staff. 
Each teacher receives a summary of the results for individual students 
in the course and the objectives of the course. The students receive 
individual reports about their own performances. A description of 
the output is available in Data Processing for Comprehensive Achievement 
Monitoring , TM-11, TM-12, and TM-14. 

Teacher and Student Reaction 

Teachers have continued to be interested and enthusiastic about 
the monitoring. They have continued into the second year of comprehen- 
sive achievement monitoring. The achievement profiles from the first 
year were used to modify and improve the course for the second year 
(TM-10| TM-16). The development of monitor items and the logistio 
problem have been the most serious obstacles to rapid implementation 
of comprehensive achievement monitoring. 

Students have reacted favorably to the new style of achievement 
testing when they see the usefulness of the reports of their results. 
They quickly become accustomed to not being able to answer every 
question on the monitor. They react positively to the display of 
their progress. 
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Dissemination 

Along with the formal means of dissemination listed below, 
several informal modes have communicated information about comprehen- 
sive achievement monitoring to a varied audience. Each of the teachers 
and principals affiliated with the project has written descriptions 
of it for and discussed it with visitors to his school. 

Technical Memoranda 
(See list on following page.) 

Professional Meetings ; Planned 

American Educational Research Association 

Symposium . Comprehensive Random Achievement Monitoring. 
Organizer: William P. Gorth, Stanford University, Stanford Center for 

Research and Development 

Chairman: David E. Evans, The University of Massachusetts 

Improving Education Using Comprehensive Random Achievement Monitoring 
Dwight W. Allen, The University of Massachusetts 
William P. Gorth, Stanford University, Stanford Center for 
Research and Development 

Instructional Objectives, Achievement Monitoring and Learning 

William P. Gorth, Stanford University, Stanford Center for 
Research and Development 

Mathematics Education Modified by Achievement Monitoring 
Lee W. Popejoy, Poway District Schools 
Instructional Management Systems Development Using CRAM 
Paul D. Pin sky 

Psychometric Issues in Comprehensive Random Achievement Monitoring 
Richard H. Linde man, Teachers College, Columbia University 
Teacher Reeducation through CRAM 

Donald H. DeLay, Davis-Macconnell-Ralston, Inc. 

Discussants: Arno Bellack, Teachers College, Columbia University 

Donald Rock, Educational Testing Service 




Technical Memoranda 



Number 

AR-1 

Th-2 

TM-3 

Tll-4 

TM-5 

Til- 6 

TM-7 

Til- 8 
TM-9 
TM-10 

TM-ll 

Tll-12 

TM-13 

TH-14 

TM-15 

Tli-16 

TM-17 

TM-18 

AR-2 



Title 



Author 



First Annual Report 

Description of courses monitored by Project CRAM 

Monitoring schedules developed for research by Project 
CRAM 



Gorth 

Gorth ft Popejoy 

Gorth, Stroud, & 
Knight 



The relation of repeated, comprehensive pretesting end Gorth, Allen, 
students 1 achievement Popejoy, & Stroud 



A comparison of comprehensive *>ersus unit pretesting and Gorth, Allen, 
students' achievement Popejoy, ft Stroud 



The evaluation of item performance In an Item sampling Llndeman, Gorth, 
case ft Allen 



Computer-Based, Instructional-testing data bank 

Separate analyses of regression 

Educational innovations monitored by Project CRAM 

Longitudinal comprehensive achievement monitoring 
in science education 



Popejoy, Gorth, 
Grayson & Stroud 

Stroud (t Gorth 

Gorth 

Gorth ft Allen 



A computer program to evaluate item performance by Gorth, Grayson 

Internal and external criteria In a longitudinal testing ft Llndeman 
program using Item sampling 



A computer program to tabulate performance profiles of Gorth. Grayson, 
longitudinal performance testing using item sampling & S croud 

Thu Project CRAM data bonk for 1967-1968 Gorth 



A computer program to compose and print tests for In- Gorth 
struct lonal testing using Item sampling 

Investigating a linear modal of learning In ninth grade Stroud ft Gorth 
algebra 



Analysis of the Project CRAM data for 1967-1968 Gorth & Plnsky 

Monitoring schedules developed for research; 1968-1969 Plnsky ft Gorth 



Demographic, aptitude, ft attitude surveys of the students, Gorth ft Plnsky 
teachers, and schools In Project CRAM 

Second annual Report to the Charles F. Kettering Allen & Gorth 
Foundation 



These reports and further Information cay be obtained by contacting the project CRAM 
staff or writing to William P. Gorth, Coordinator, project CRAM, School of Education, 
Stanford University, Stanford, California, 94105 
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National Council on Heasurenent In Education 

Paper . Item Analysis in an Item Sampling Casei by R, H. 

Lindeman, W. P. Oorth, and D.'W. Allen. 

National Soienoe Teachers Association 

Paper . Longitudinal Comprehensive Achievement Monitoring in 
Science Education: Course evaluation and individual diagnosis, by 

W. P. Oorth and D. W. Allen. 

National Council of Teachers of Mathematics 

Paper . Longitudinal Comprehensive Achievement Monitoring in 
Mathematics Education, by W, P. Oorth and D. W. Allen 

Professional Journals; Planned 

The relation of repeated, comprehensive pretesting and students* 
achievement, by Oorth, Allen, Popejoy, & Stroud. 

A comparison of comprehensive versus unit pretesting and 
students* achievement, by Oorth, Allen, Popejoy, & Stroud. 

The evaluation of item performance in an item sampling case, by 
Lindeman, Qorth, & Allen. 

Computer-based, instructional-testing data bank, by Pope joy, 
Oorth, Orayson & Stroud. 

Educational innovations monitored by Project CRAM, by Oorth. 
Longitudinal comprehensive achievement monitoring in science 
education, by Oorth fr Allen. 

A computer program to evaluate item performance by internal and 
external criteria in a longitudinal testing program using item sampling, 
by Oor*h, Orayson, & Lindeman. 

A computer program to tabulate performance profiles of longi- 
tudinal performance testing using item sampling, by Oorth, Grayson, 

& Stroud. 

A computer program to compose and print tests for instructional 
testing U6ing item sampling, by Oorth. 

Investigating a linear model of learning in ninth grade algebra, 
by Stroud & Qorth. 
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Priorities 

(1) Develop revision of teacher-based comprehensive achievement 
monitoring. 

(2) Incorporate comprehensive achievement monitoring 
Into teacher education program. 

(3) Increase objective and item banks. 

(4) Reduce time needed to set up monitoring programs in courses. 

(3) Tell many more people about comprehensive achievement monitoring. 

References 

(See the following two pages.) 
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Table TK-I7.1a Summary of Achievement Monitoring 
Schedules, 1968 1969 



u 

w 

d 




"O 

O 


4 


s 


It 

<u 




1 


£ 

0 




§ 


u 




fj 


a 


c 


«A 4.> 


C 


o: v> 


W Vi 


YJ 


v> 


f 


0 


M 


23 


c 


<U ^ 


c 


d -ri 


g -h 


C* 


<H 


Post 




a 

rV 

P-« 


« 

PV 


d 

*v 


Jh 

v£> 

OP 


c 

r-« 


^£> 

CP 


>* 

fs 

CP 




o 

CP 



4> 

U 

S 



W 

w 5 
a) g 

^ 4J 

& ^ 



o 

u* 

V 
1) 
O 



M 

(A g 
* N 

r*i *h 

r/} 



O 



c 

C\J 



O 

CM 



<7% 



O 

iT* 



O 



o 

<N 



O 



c? 


0 










c 


il 

O 


fri3 


frJ2 


v ^ 






3 


4 3 


g S 


g 3- 


g 3 


oo 

g 


X? 

g 


> 

CU UP 


w 


k' 

W Mf 


dtl <s3" 


H 


Vi 












Px- 















£ 


a 


£ 


g 




g 


s 


g 


u 




« 


W Vi 


(A Vi 


g 


C 


a *h 


O *H 


g -h 


>< 


V2 




£*« 




QO 




*£> 


<M 


o 


H 




cp 


OP 


CP 



O' 



rP 

M ^ 
4> 4) 

^ V 
M & 



<■ 

CM 



up 



•?i K 

v A' 

4} ? 

W cv 



4> 

w <t 



3 



<r 



e> 



>» « 
*< r* 

o y 



9 


o 


o 


O 


O 


O 


“V 


rV 


cp 


CO 

<S! 


<P 


1. > 


r->* 


1-4 


ri 


r-i 


r-4 



o 

r-> 



&5 

S * 

W cn 



o 

4- 1 



g 


! 3 


o 


r< 




a 


CM 


O 


CM 


U“» 


V. 


j r-4 


r**i 


<M 




CM 




<! 




o 


Vt 

3 


i 


CN 


<o 


o 


<r 


' J 


CN 




CM 


O , 

o : 


) PK 
: ^ 


ri 


• V< 

r; 


r> 


y> 




< 


tl 


ri 

r-* 



o 

ERIC 






jo 

CO 

H 



C 

0 

no -ri 

0 p 

^ ^ o 
wop 
g p 

>2 



p 

g 

p 

P 4) 

21 

M 

p O 
U 

0 'O 



3 

CO 




M 



*C 

p 

o 



6 

•8 

| 

AS 



M 

H 



P 

O 







P 


M 

M 


S* 


p 






o *o 




g x> 


g tj 






,0 <U 


no 


,c « 


,e u 


M 


M 


o o 


4) 


u u 


o u 


M 


M 


«tf *6 


v-( 


• c 


O <0 






g a* 




g P 


g Ai 






H 




H 


H 



•8 

a 



o 



P 



*-< 






•c 


■0 




r-i 








p 


p 


,n 




«. 


* 


•>. 


o 


1—4 


p 


•» 


« 


u 


>> 

60 


r-l 


(Ml 


o> 


& 


■H 


o 


o 


» 


#, 


* 


o 


rH 


u 


r-i 


rC 




& 


P 


to 


w 


O 


p 


p 


P 


10 


0 


•H 


♦rC 


* 


« 


n 

>'* 


•rC 


W 


w 


P 


£ 


a 


0G 



,c 

P 

CM 

»P 

<£ 



P 

a 



no 

s 

P« 

CO 



g 

c 

s§ 



g 

3 

J§ 



60 


0 


c 


c 


c 


60 

< 

P 


0 g 


O 


o 


0 


o 


g 


v<< M 


•ri 


•H 




*ri 


g > 


rP 3 


U 


p 


P 


p 


•t; < 


A *& 


u 


u 


U 


u 


< e 


no g 


g 


g 


g 


g 


p p 


8 g 


CO 


in 


CO 


CO 


O 0 


u 8 




>> 






^ o 


CO p 


p 


» 


w 


P 


PO A< 



p 
•N 
r-i 
P ^4 

g x> 

rC < 

u 

«tf %> 
o? o 
H co 
*a 
►> 3 
« 



cn 

0 




H 



P 

O 



« 



O 




o 








> 




p 








*P 




‘H 








CO 




to 






XJ 


w 


g 


» 


g 


O 


o 


o 


c 


g 


c 


0 


o 


u 


o 


o 


o 


O 


g 


o 


»TT 


o 


53 


55 


CL 


3 




3 






CO 


CO 




to 









g 

60 

« 



c 


H 


0 


o 


g 


o 


«p 


g > 


*w 


p 


rq < 


p 


g 


•C 


o 


g 


M P 


0 




°.s 


CO 




>> 0 


>> 


CQ 


PQ Ai 


CO 



« 

g 

*8 "So 

l| 

*0 d 

c o 
s £ 



no 

0 

4 



g 


o 


o 




O 


o 


CM 


o 


CM 


yr\ 


w 


1— i 


*p 


CM 


r-i 


CM 


O 


<* 




O 


p 

3 

O 

o 


r-i 


CM 




<r 


't 




CM 




CM 


§ 




§ 


M 

P 


2 


n 




$ 


d 

#C4 



O 

ERIC 



